{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e06275b8",
   "metadata": {},
   "source": [
    "# How to Build a Video Copy Detection Engine\n",
    "\n",
    "This notebook illustrates how to build a video copy detection engine from scratch using [Milvus](https://milvus.io/) and [Towhee](https://towhee.io/).\n",
    "\n",
    "\n",
    "**What is Video Copy Detection?**\n",
    "\n",
    "Video Copy Detection, also known as Video Identification by Fingerprinting, is to retrieve the similar or exact same video for a given query video. \n",
    "\n",
    "Due to the popularity of Internet-based video sharing services, the volume of video content on the Internet has reached unprecedented scales. A video copy detection system is important in applications like video classification, tracking, filtering and recommendation, not to mention the field of copyright protection.  \n",
    "\n",
    "However, content-based video retrieval is particularly hard in practice, one needs to calculate the similarity between the given video and each and every video in a database to retrieve and rank similar ones based on relevance. Threfore, We hereby introduce Milvus and Towhee to help building a Video Deduplication System within several lines.\n",
    "\n",
    "**What are Milvus & Towhee?**\n",
    "\n",
    "- Milvus is the most advanced open-source vector database built for AI applications and supports nearest neighbor embedding search across tens of millions of entries.\n",
    "- Towhee is a framework that provides ETL for unstructured data using SoTA machine learning models.\n",
    "\n",
    "We'll go through video retrieval procedures and evaluate the performance. Moreover, we managed to make the core functionality as simple as few lines of code, with which you can start hacking your own video copy detection engine."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d953a08",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "### Install packages\n",
    "\n",
    "Make sure you have installed required python packages:\n",
    "\n",
    "| package |\n",
    "| -- |\n",
    "| towhee |\n",
    "| pillow |\n",
    "| ipython |\n",
    "| pandas |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "469282e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q towhee pillow ipython pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4f5a71c",
   "metadata": {},
   "source": [
    "### Prepare the data\n",
    "\n",
    "First, we need to prepare the dataset and Milvus environment.   \n",
    "\n",
    "[VCDB:A Large-Scale Database for Partial Copy Detection in Videos](https://fvl.fudan.edu.cn/dataset/vcdb/list.htm) is a popular dataset for video deduplication task. It contains over 100,000 Web videos, and more than 9,000 copied segment pairs found through careful manual annotation.\n",
    "\n",
    "VCDB consists of two parts: the core dataset and the background dataset. The core dataset (528 videos, approximately 27 hours) was collected using 28 carefully selected queries from YouTube and MetaCafe. After extensive manual annotation, 9,236 pairs of partial copies were found. Major transformations between the copies include \"insertion of patterns\", \"camcording\", \"scale change\", \"picture in picture\", etc. \n",
    "\n",
    "In this tutorial, we prepare a subset of VCDB core dataset, which contains 20 events, and each of them contains about 5 videos with the same or similar content. This takes about 1.3G of space.\n",
    "\n",
    "Let's take a quick look"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "110c3cf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "22543226",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "100 1357M  100 1357M    0     0  4823k      0  0:04:48  0:04:48 --:--:-- 8655k519k      0  0:05:07  0:01:00  0:04:07 6986k  0  4581k      0  0:05:03  0:01:02  0:04:01 6921k4:55  0:01:05  0:03:50 7049k 5258k      0  0:04:24  0:01:55  0:02:29 4959k 5369k      0  0:04:18  0:02:02  0:02:16 7525k291k      0  0:04:22  0:02:13  0:02:09 4133k     0  5364k      0  0:04:19  0:02:35  0:01:44 5934k     0  0:04:15  0:02:41  0:01:34 7470k 0     0  5086k      0  0:04:33  0:03:16  0:01:17 1963k   0  0:04:44  0:03:34  0:01:10 4479k  0  0:04:47  0:03:51  0:00:56 3882k0:00:44 3168k6 3414k0  0:04:57  0:04:32  0:00:25 3567k 0  0:04:50  0:04:45  0:00:05 8669k 4816k      0  0:04:48  0:04:47  0:00:01 8841k\n"
     ]
    }
   ],
   "source": [
    "! curl -L https://github.com/towhee-io/examples/releases/download/data/VCDB_core_sample.zip -O\n",
    "! unzip -q -o VCDB_core_sample.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "091912bc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>path</th>\n",
       "      <th>event</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>VCDB_core_sample/mr_and_mrs_smith_tango/e2adc7...</td>\n",
       "      <td>mr_and_mrs_smith_tango</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>VCDB_core_sample/mr_and_mrs_smith_tango/be5178...</td>\n",
       "      <td>mr_and_mrs_smith_tango</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>VCDB_core_sample/mr_and_mrs_smith_tango/9ac82e...</td>\n",
       "      <td>mr_and_mrs_smith_tango</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>VCDB_core_sample/mr_and_mrs_smith_tango/e8aa83...</td>\n",
       "      <td>mr_and_mrs_smith_tango</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>VCDB_core_sample/mr_and_mrs_smith_tango/bf5822...</td>\n",
       "      <td>mr_and_mrs_smith_tango</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>VCDB_core_sample/the_legend_of_1900_magic_walt...</td>\n",
       "      <td>the_legend_of_1900_magic_waltz</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>VCDB_core_sample/the_legend_of_1900_magic_walt...</td>\n",
       "      <td>the_legend_of_1900_magic_waltz</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>VCDB_core_sample/the_legend_of_1900_magic_walt...</td>\n",
       "      <td>the_legend_of_1900_magic_waltz</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>VCDB_core_sample/the_legend_of_1900_magic_walt...</td>\n",
       "      <td>the_legend_of_1900_magic_waltz</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>VCDB_core_sample/the_legend_of_1900_magic_walt...</td>\n",
       "      <td>the_legend_of_1900_magic_waltz</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                path  \\\n",
       "0  VCDB_core_sample/mr_and_mrs_smith_tango/e2adc7...   \n",
       "1  VCDB_core_sample/mr_and_mrs_smith_tango/be5178...   \n",
       "2  VCDB_core_sample/mr_and_mrs_smith_tango/9ac82e...   \n",
       "3  VCDB_core_sample/mr_and_mrs_smith_tango/e8aa83...   \n",
       "4  VCDB_core_sample/mr_and_mrs_smith_tango/bf5822...   \n",
       "5  VCDB_core_sample/the_legend_of_1900_magic_walt...   \n",
       "6  VCDB_core_sample/the_legend_of_1900_magic_walt...   \n",
       "7  VCDB_core_sample/the_legend_of_1900_magic_walt...   \n",
       "8  VCDB_core_sample/the_legend_of_1900_magic_walt...   \n",
       "9  VCDB_core_sample/the_legend_of_1900_magic_walt...   \n",
       "\n",
       "                            event  id  \n",
       "0          mr_and_mrs_smith_tango   1  \n",
       "1          mr_and_mrs_smith_tango   2  \n",
       "2          mr_and_mrs_smith_tango   3  \n",
       "3          mr_and_mrs_smith_tango   4  \n",
       "4          mr_and_mrs_smith_tango   5  \n",
       "5  the_legend_of_1900_magic_waltz   6  \n",
       "6  the_legend_of_1900_magic_waltz   7  \n",
       "7  the_legend_of_1900_magic_waltz   8  \n",
       "8  the_legend_of_1900_magic_waltz   9  \n",
       "9  the_legend_of_1900_magic_waltz  10  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "from pathlib import Path\n",
    "import torch\n",
    "import pandas as pd\n",
    "random.seed(6)\n",
    "\n",
    "root_dir = './VCDB_core_sample'\n",
    "\n",
    "\n",
    "min_sample_num = 5\n",
    "sample_folder_num = 20\n",
    "\n",
    "all_video_path_lists = []\n",
    "all_video_path_list = []\n",
    "\n",
    "df = pd.DataFrame(columns=('path','event','id'))\n",
    "query_df = pd.DataFrame(columns=('path','event','id'))\n",
    "\n",
    "video_idx = 0\n",
    "for i, mid_dir_path in enumerate(Path(root_dir).iterdir()):\n",
    "    if i >= sample_folder_num:\n",
    "        break\n",
    "    if mid_dir_path.is_dir():\n",
    "        path_videos = list(Path(mid_dir_path).iterdir())\n",
    "        if len(path_videos) < min_sample_num:\n",
    "            print('len(path_videos) < min_sample_num, continue.')\n",
    "            continue\n",
    "        sample_video_path_list = random.sample(path_videos, min_sample_num)\n",
    "        all_video_path_lists.append(sample_video_path_list)\n",
    "        all_video_path_list += [str(path) for path in sample_video_path_list]\n",
    "        for j, path in enumerate(sample_video_path_list):\n",
    "            video_idx += 1\n",
    "            if j == 0:\n",
    "                query_df = pd.concat([query_df, pd.DataFrame({'path': [str(path)],'event':[path.parent.stem],'id': [str(video_idx)]})], ignore_index=True)\n",
    "            df = pd.concat([df, pd.DataFrame({'path': [str(path)],'event':[path.parent.stem],'id': [str(video_idx)]})], ignore_index = True)\n",
    "\n",
    "all_sample_video_dicts = []\n",
    "for i, sample_video_path_list in enumerate(all_video_path_lists):\n",
    "    anchor_video = sample_video_path_list[0]\n",
    "    pos_video_path_list = sample_video_path_list[1:]\n",
    "    neg_video_path_lists = all_video_path_lists[:i] + all_video_path_lists[i + 1:]\n",
    "    neg_video_path_list = [neg_video_path_list[0] for neg_video_path_list in neg_video_path_lists]\n",
    "    all_sample_video_dicts.append({\n",
    "        'anchor_video': anchor_video,\n",
    "        'pos_video_path_list': pos_video_path_list,\n",
    "        'neg_video_path_list': neg_video_path_list\n",
    "    })\n",
    "\n",
    "id2event = df.set_index(['id'])['event'].to_dict()\n",
    "id2path = df.set_index(['id'])['path'].to_dict()\n",
    "\n",
    "df_csv_path = 'video_info.csv'\n",
    "query_df_csv_path = 'query_video_info.csv'\n",
    "df.to_csv(df_csv_path)\n",
    "query_df.to_csv(query_df_csv_path)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef4670d6",
   "metadata": {},
   "source": [
    "Define some helper function to convert video to gif so that we can have a look at these videos.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3ea961d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "from pathlib import Path\n",
    "import towhee\n",
    "from towhee import pipe, ops\n",
    "from PIL import Image\n",
    "\n",
    "def display_gif(video_path_list, text_list):\n",
    "    html = ''\n",
    "    for video_path, text in zip(video_path_list, text_list):\n",
    "        html_line = '<img src=\"{}\"> {} <br/><br/>'.format(video_path, text)\n",
    "        html += html_line\n",
    "    return display.HTML(html)\n",
    "\n",
    "    \n",
    "def convert_video2gif(video_path, output_gif_path, num_samples=16):\n",
    "    p = (\n",
    "        pipe.input('video_file')\n",
    "        .flat_map('video_file', 'frame', ops.video_decode.ffmpeg(start_time=0.0, end_time=1000.0, sample_type='time_step_sample', args={'time_step': 5}))\n",
    "        .output('frame')\n",
    "    )\n",
    "    frames = p(video_path).to_list()\n",
    "    imgs = [Image.fromarray(frame[0]) for frame in frames]\n",
    "    imgs[0].save(fp=output_gif_path, format='GIF', append_images=imgs[1:], save_all=True, loop=0)\n",
    "\n",
    "\n",
    "def display_gifs_from_video(video_path_list, text_list, tmpdirname = './tmp_gifs'):\n",
    "    Path(tmpdirname).mkdir(exist_ok=True)\n",
    "    gif_path_list = []\n",
    "    for video_path in video_path_list:\n",
    "        video_name = str(Path(video_path).name).split('.')[0]\n",
    "        gif_path = Path(tmpdirname) / (video_name + '.gif')\n",
    "        convert_video2gif(video_path, gif_path)\n",
    "        gif_path_list.append(gif_path)\n",
    "    return display_gif(gif_path_list, text_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97440442",
   "metadata": {},
   "source": [
    "Positive denotes a video that is contain same content event in anchor video, while negative denotes an inconsistent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "88519061",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['anchor video: t-mac_13_points_in_35_seconds', 'positive video 1', 'negative video 1: mr_and_mrs_smith_tango']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"tmp_gifs/b61905d41276ccf2af59d4985158f8b1ce1d4990.gif\"> anchor video: t-mac_13_points_in_35_seconds <br/><br/><img src=\"tmp_gifs/2bdf8029b38735a992a56e32cfc81466eea81286.gif\"> positive video 1 <br/><br/><img src=\"tmp_gifs/e2adc784b83446ae775f698b9d17c9fd392b2f75.gif\"> negative video 1: mr_and_mrs_smith_tango <br/><br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_video_pair = random.sample(all_sample_video_dicts, 1)[0]\n",
    "neg_sample_num = min(1, sample_folder_num)\n",
    "anchor_video = random_video_pair['anchor_video']\n",
    "anchor_video_event = anchor_video.parent.stem\n",
    "pos_video_list = random_video_pair['pos_video_path_list'][:1]\n",
    "pos_video_list_events = [path.parent.stem for path in pos_video_list][:1]\n",
    "neg_video_list = random_video_pair['neg_video_path_list'][:neg_sample_num]\n",
    "neg_video_list_events = [path.parent.stem for path in neg_video_list]\n",
    "\n",
    "show_video_list = [str(anchor_video)] + [str(path) for path in pos_video_list] + [str(path) for path in neg_video_list]\n",
    "# print(show_video_list)\n",
    "caption_list = ['anchor video: ' + anchor_video_event] + ['positive video ' + str(i + 1) for i in range(len(pos_video_list))] + ['negative video ' + str(i + 1) + ': ' + neg_video_list_events[i] for i in range(len(neg_video_list))]\n",
    "print(caption_list)\n",
    "tmpdirname = './tmp_gifs'\n",
    "display_gifs_from_video(show_video_list, caption_list, tmpdirname=tmpdirname)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c707302",
   "metadata": {},
   "source": [
    "### Create a Milvus Collection\n",
    "\n",
    "Before getting started, please make sure that you have started a [Milvus service](https://milvus.io/docs/install_standalone-docker.md). This notebook uses [milvus 2.2.10](https://milvus.io/docs/v2.2.x/install_standalone-docker.md) and [pymilvus 2.2.11](https://milvus.io/docs/release_notes.md#2210)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0aa74ab2",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q pymilvus==2.2.11"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd870876",
   "metadata": {},
   "source": [
    "Let's first create a `video deduplication` collection that uses the [L2 distance metric](https://milvus.io/docs/metric.md#Euclidean-distance-L2) and an [IVF_FLAT index](https://milvus.io/docs/index.md#IVF_FLAT)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "df83801e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility\n",
    "\n",
    "connections.connect(host='127.0.0.1', port='19530')\n",
    "\n",
    "def create_milvus_collection(collection_name, dim):\n",
    "    if utility.has_collection(collection_name):\n",
    "        utility.drop_collection(collection_name)\n",
    "    \n",
    "    fields = [\n",
    "    FieldSchema(name='id', dtype=DataType.VARCHAR, descrition='ids',max_length=500, is_primary=True, auto_id=False),\n",
    "    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, descrition='embedding vectors', dim=dim)\n",
    "    ]\n",
    "    schema = CollectionSchema(fields=fields, description='video deduplication')\n",
    "    collection = Collection(name=collection_name, schema=schema)\n",
    "\n",
    "    # create IVF_FLAT index for collection.\n",
    "    index_params = {\n",
    "        'metric_type':'L2', #IP\n",
    "        'index_type':\"IVF_FLAT\",\n",
    "        'params':{\"nlist\":2048}\n",
    "    }\n",
    "    collection.create_index(field_name=\"embedding\", index_params=index_params)\n",
    "    return collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dcfe3988",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = create_milvus_collection('video_deduplication', 1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7acaf86b",
   "metadata": {},
   "source": [
    "## Video Copy Detection\n",
    "\n",
    "In this section, we'll show how to build our Video Copy Detection engine using Milvus. The basic idea behind Video Copy Detection is the extract embeddings from videos using Deep Neural Network and store them in Milvus, then get query videos embeddings and compare with those stored in Milvus.\n",
    "\n",
    "We use [Towhee](https://towhee.io/), a machine learning framework that help users to build data processing pipelines. [Towhee](https://towhee.io/) also provides pre-defined operators which implement insert and query operation in Milvus.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fdcd9b8",
   "metadata": {},
   "source": [
    "### Load Video Embeddings into Milvus\n",
    "\n",
    "We first extract embeddings from images and insert the embeddings into Milvus for indexing. Towhee provides a [method-chaining style API](https://towhee.readthedocs.io/en/main/index.html) so that users can assemble a data processing pipeline with operators.   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1dc07d2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 7s, sys: 34.4 s, total: 3min 42s\n",
      "Wall time: 2min 49s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import os\n",
    "from csv import reader\n",
    "\n",
    "import towhee\n",
    "from towhee.datacollection import DataCollection\n",
    "from towhee import pipe, ops\n",
    "device = 'cuda:1'\n",
    "# device = 'cpu'\n",
    "\n",
    "emb_pipe = (\n",
    "    pipe.input('path', 'event', 'id')\n",
    "        .flat_map('path', 'frames', ops.video_decode.ffmpeg(start_time=0.0, end_time=60.0, sample_type='time_step_sample', args={'time_step': 1}))\n",
    "        .window_all('frames', 'video', lambda x: x)\n",
    "        .map('video', 'emb', ops.video_copy_detection.distill_and_select(model_name='cg_student', device = device))\n",
    "        .map(('id', 'emb'), 'insert_res', ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='video_deduplication'))\n",
    "        .output()\n",
    ")\n",
    "\n",
    "with open(df_csv_path, 'r') as f:\n",
    "    csv_reader = reader(f)\n",
    "    _ = next(csv_reader)\n",
    "    for n, row in enumerate(csv_reader):\n",
    "        res = emb_pipe(*row[1:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f96e5c4b",
   "metadata": {},
   "source": [
    "Here is detailed explanation for each line of the code:\n",
    "\n",
    "- `ops.video_decode.ffmpeg(start_time=0.0, end_time=60.0, sample_type='time_step_sample', args={'time_step': 1})`: Subsample and decode the first 60 seconds of the video to get a list of frames, one frame per second.\n",
    "\n",
    "- `window_all('frames', 'video', lambda x: x)`: Append all frames into a list.\n",
    "\n",
    "- `ops.video_copy_detection.distill_and_select(model_name='cg_student', device = device)`: Extract embedding from the frames using Coarse Grained Student model in DnS.\n",
    "\n",
    "- `ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='video_deduplication')`: Insert video embedding in to Milvus collection `video_deduplication`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "97e3f3f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of inserted data is 95.\n"
     ]
    }
   ],
   "source": [
    "print('Total number of inserted data is {}.'.format(collection.num_entities))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4af2a4e3",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We have finished the core functionality of the Video Copy Detection engine. However, we don't know whether it achieves a reasonable performance. We need to evaluate the retrieval engine against the ground truth.\n",
    "\n",
    "In this section, we'll evaluate the strength of our text-video retrieval using mAP@topk:   \n",
    "`mAP@topk` is the proportion of relevant items found in the top-k recommendations. Suppose that we computed precision at 10 examples and found it is 40% in our top-10 recommendation system. This means that 40% of the recall examples are real positive ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "81b8b5ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "video VCDB_core_sample/scent_of_woman_tango/ac1543adad8ecc32d98d9a67d75fb0a39f67b685.flv's top 5 search result: ['VCDB_core_sample/scent_of_woman_tango/ac1543adad8ecc32d98d9a67d75fb0a39f67b685.flv', 'VCDB_core_sample/scent_of_woman_tango/f3c8c0c9b93e0a49d2508eee4aae618c1d69e082.flv', 'VCDB_core_sample/scent_of_woman_tango/c29e02d9e847bcde39ee667b454e4d67b22b105c.flv', 'VCDB_core_sample/scent_of_woman_tango/d1a19730dcc2b5ea5bec61756c452772aae031dd.flv', 'VCDB_core_sample/beautiful_mind_game_theory/6171d3d87ae377e497199554033bca96a263277b.mp4']\n",
      "video VCDB_core_sample/kennedy_assassination_slow_motion/71ba3405d1e896d04c89754ff618b4198d2e65d5.flv's top 5 search result: ['VCDB_core_sample/kennedy_assassination_slow_motion/71ba3405d1e896d04c89754ff618b4198d2e65d5.flv', 'VCDB_core_sample/kennedy_assassination_slow_motion/6a2f5ceaa65dd1187ff57ff1c1fccee9da91985b.flv', 'VCDB_core_sample/kennedy_assassination_slow_motion/acdf1a72b8cf37ee2609cdc03ff70747f36e0476.flv', 'VCDB_core_sample/kennedy_assassination_slow_motion/f6e296bfd930f585e165246530efa97bc035330f.flv', 'VCDB_core_sample/kennedy_assassination_slow_motion/8a268ef5ad1f250fca37321ba2e9c9224b084dba.flv']\n",
      "video VCDB_core_sample/troy_achilles_and_hector/ccc879ecfb35a1a77667dd8357b71a930c19092c.flv's top 5 search result: ['VCDB_core_sample/troy_achilles_and_hector/ccc879ecfb35a1a77667dd8357b71a930c19092c.flv', 'VCDB_core_sample/troy_achilles_and_hector/ee417a6b882853ffcd3f78b380b0205a9411f4d6.flv', 'VCDB_core_sample/troy_achilles_and_hector/a89a3193db3354c059dfe4effac05c4667f9c239.flv', 'VCDB_core_sample/troy_achilles_and_hector/6fe097a963673b26c62f6ff6d6151d383c194b9d.flv', 'VCDB_core_sample/troy_achilles_and_hector/0b3f9e88e5ab73e19dc4d1a32115ea3457867128.flv']\n",
      "video VCDB_core_sample/maradona_hand_of_god/3ec7ab68ed6a1690d7c0074bc563d0dccac57726.flv's top 5 search result: ['VCDB_core_sample/maradona_hand_of_god/3ec7ab68ed6a1690d7c0074bc563d0dccac57726.flv', 'VCDB_core_sample/david_beckham_lights_the_olympic_torch/54c560a6b270647b28ea2746853c937885e2718b.flv', 'VCDB_core_sample/maradona_hand_of_god/5965d78e520ef5c162f22da88745f4c6dccd6fcf.flv', 'VCDB_core_sample/run_forrest_fun/1e5aec100a5b897342a4be9246995fc16f784e66.flv', 'VCDB_core_sample/scent_of_woman_tango/ddf4ea3ad374dad0d4abc418418581632547f2fc.flv']\n",
      "video VCDB_core_sample/run_forrest_fun/1777dc7aada63176bc9d3d87a5ce322163cfe4e6.flv's top 5 search result: ['VCDB_core_sample/run_forrest_fun/1777dc7aada63176bc9d3d87a5ce322163cfe4e6.flv', 'VCDB_core_sample/run_forrest_fun/7ac8224504f430258be5cf7c3e21bd2611fa7c77.flv', 'VCDB_core_sample/run_forrest_fun/8f19329946455ae5c2c7b788ea6f6513bf5e1c9a.flv', 'VCDB_core_sample/troy_achilles_and_hector/a89a3193db3354c059dfe4effac05c4667f9c239.flv', 'VCDB_core_sample/kennedy_assassination_slow_motion/acdf1a72b8cf37ee2609cdc03ff70747f36e0476.flv']\n",
      "video VCDB_core_sample/obama_kicks_door/1f73466d86f0a92140bd2f89eb95e8c147b47935.flv's top 5 search result: ['VCDB_core_sample/obama_kicks_door/1f73466d86f0a92140bd2f89eb95e8c147b47935.flv', 'VCDB_core_sample/david_beckham_lights_the_olympic_torch/09b682c899b0727e9990d8e347cdce3df7c5550e.flv', 'VCDB_core_sample/troy_achilles_and_hector/0b3f9e88e5ab73e19dc4d1a32115ea3457867128.flv', 'VCDB_core_sample/scent_of_woman_tango/d1a19730dcc2b5ea5bec61756c452772aae031dd.flv', 'VCDB_core_sample/troy_achilles_and_hector/ee417a6b882853ffcd3f78b380b0205a9411f4d6.flv']\n",
      "video VCDB_core_sample/mr_and_mrs_smith_tango/9ac82ed37390f45bdc81b699c53085b29143ee70.flv's top 5 search result: ['VCDB_core_sample/mr_and_mrs_smith_tango/9ac82ed37390f45bdc81b699c53085b29143ee70.flv', 'VCDB_core_sample/mr_and_mrs_smith_tango/bf582249cfc79d691195a8681961029cc5149a76.flv', 'VCDB_core_sample/beautiful_mind_game_theory/5f6527456bf2b4c460737a1c222dbd0e70009030.flv', 'VCDB_core_sample/beautiful_mind_game_theory/b41c61fb756014444b7974ef25f2c76d5c3b42e4.flv', 'VCDB_core_sample/mr_and_mrs_smith_tango/be51784c0b0676bf487376af362b088922efa954.flv']\n",
      "video VCDB_core_sample/the_legend_of_1900_magic_waltz/36058f2303f8f3792127c3aba23a770e2ce3a2af.flv's top 5 search result: ['VCDB_core_sample/the_legend_of_1900_magic_waltz/36058f2303f8f3792127c3aba23a770e2ce3a2af.flv', 'VCDB_core_sample/the_legend_of_1900_magic_waltz/b90b0fdb3035754c78e100577387be7c39ea2404.flv', 'VCDB_core_sample/the_legend_of_1900_magic_waltz/500bc7f5c5c61a19a8d4489fece07fdc1d7ed50e.flv', 'VCDB_core_sample/the_legend_of_1900_magic_waltz/4849d1042b92a04fb2e019beec04a4f30188ba18.flv', 'VCDB_core_sample/the_legend_of_1900_magic_waltz/50dfd1daee780fbf9722990387201e2b367a1f27.flv']\n",
      "video VCDB_core_sample/the_last_samurai_last_battle/5f8d7efc28c6dd3e20cfd0060177edcb243a0456.flv's top 5 search result: ['VCDB_core_sample/the_last_samurai_last_battle/5f8d7efc28c6dd3e20cfd0060177edcb243a0456.flv', 'VCDB_core_sample/the_last_samurai_last_battle/93cad7675cea2f7c761ad9fc2ee72baee61f38e3.flv', 'VCDB_core_sample/the_last_samurai_last_battle/4a84b3090139a049a0dfd9cd4ae6db8b446fec99.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/743b4b30bf4fa11e3231dd0cc319016db1cb5b73.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/283043f1c1433d9205416d2922764421d3d7e34e.flv']\n",
      "video VCDB_core_sample/president_obama_takes_oath/66246823d79897fd7d93677c6c0244a3cbd4657f.flv's top 5 search result: ['VCDB_core_sample/president_obama_takes_oath/66246823d79897fd7d93677c6c0244a3cbd4657f.flv', 'VCDB_core_sample/president_obama_takes_oath/54b8c1ae26a16400c15badbae407c809b3ebf03c.flv', 'VCDB_core_sample/president_obama_takes_oath/1d4062ded72a30caa898d2bb9c105994c8a96acc.flv', 'VCDB_core_sample/president_obama_takes_oath/e29e65d0e362b8e7d450d833227ea3c0f5f65f12.flv', 'VCDB_core_sample/president_obama_takes_oath/4120e70bc6332bd6010e76b0054c45653bfa6f94.flv']\n",
      "video VCDB_core_sample/bolt_beijing_100m/5b1b41404d1bad5d231bd8c7fdbae109e0ea1fc2.flv's top 5 search result: ['VCDB_core_sample/bolt_beijing_100m/5b1b41404d1bad5d231bd8c7fdbae109e0ea1fc2.flv', 'VCDB_core_sample/bolt_beijing_100m/d30e718464711d5903b53d36aef8c8212bad1d07.flv', 'VCDB_core_sample/bolt_beijing_100m/359b3ef99acac4136096052c6d812a6248811e10.flv', 'VCDB_core_sample/bolt_beijing_100m/098df9c477ecf6c4f3818b198490b0258163015d.flv', 'VCDB_core_sample/bolt_beijing_100m/225eceebd0f24728bd1740f642de6705d43a4da0.flv']\n",
      "video VCDB_core_sample/dove_evolution_commercial/46fa74f0aa8e67c49c64802317c4de3894f0e1b9.flv's top 5 search result: ['VCDB_core_sample/dove_evolution_commercial/46fa74f0aa8e67c49c64802317c4de3894f0e1b9.flv', 'VCDB_core_sample/dove_evolution_commercial/8870293c58fbed9cc03a738577e5802164199d8b.flv', 'VCDB_core_sample/dove_evolution_commercial/8fb97b98e2f27c9f162d6c1d6c27806c84af8e8d.flv', 'VCDB_core_sample/dove_evolution_commercial/bbdbe357f408fb84d5f57c1522a90504f3162d6f.flv', 'VCDB_core_sample/mr_and_mrs_smith_tango/be51784c0b0676bf487376af362b088922efa954.flv']\n",
      "video VCDB_core_sample/baggio_penalty_1994/37b31d607d31a47d347b15dae2b8aa63e57861eb.flv's top 5 search result: ['VCDB_core_sample/baggio_penalty_1994/37b31d607d31a47d347b15dae2b8aa63e57861eb.flv', 'VCDB_core_sample/baggio_penalty_1994/6d1a89c83d554fc6a5e39fcadb172a79baf140fd.mp4', 'VCDB_core_sample/baggio_penalty_1994/3504e360accbaccb1580befbb441f1019664c2bb.mp4', 'VCDB_core_sample/baggio_penalty_1994/6d1466ebc4de7e5ddb229bde090b5c5acac15c0c.flv', 'VCDB_core_sample/bolt_beijing_100m/098df9c477ecf6c4f3818b198490b0258163015d.flv']\n",
      "video VCDB_core_sample/t-mac_13_points_in_35_seconds/3d0a3002441f682c7124806eb9b92c677af2ee9e.flv's top 5 search result: ['VCDB_core_sample/t-mac_13_points_in_35_seconds/3d0a3002441f682c7124806eb9b92c677af2ee9e.flv', 'VCDB_core_sample/t-mac_13_points_in_35_seconds/2bdf8029b38735a992a56e32cfc81466eea81286.flv', 'VCDB_core_sample/t-mac_13_points_in_35_seconds/b61905d41276ccf2af59d4985158f8b1ce1d4990.flv', 'VCDB_core_sample/t-mac_13_points_in_35_seconds/e4b443e64c27a3364d16db8e11e6e85f2d3fd7ed.flv', 'VCDB_core_sample/t-mac_13_points_in_35_seconds/5df28e18b3d8fbdc0f4cd07ef5aefcdc1b4f8d42.flv']\n",
      "video VCDB_core_sample/ronaldinho_ping_pong/ab2879d4e100749bb69b5892c86778853022af77.flv's top 5 search result: ['VCDB_core_sample/ronaldinho_ping_pong/275ca13a391d89d08b7de46c23c4430070c3251c.flv', 'VCDB_core_sample/ronaldinho_ping_pong/ab2879d4e100749bb69b5892c86778853022af77.flv', 'VCDB_core_sample/ronaldinho_ping_pong/980970c70070421c46fb122e7a6ebc59682e697d.flv', 'VCDB_core_sample/ronaldinho_ping_pong/4f1435d832285500d7663d893b9eeba69262321a.flv', 'VCDB_core_sample/ronaldinho_ping_pong/5567a1bb5b9767342dc99baea8d67508ca22c046.flv']\n",
      "video VCDB_core_sample/saving_private_ryan_omaha_beach/200130e0c39f0239a22ad3f873feac30e88134df.flv's top 5 search result: ['VCDB_core_sample/saving_private_ryan_omaha_beach/200130e0c39f0239a22ad3f873feac30e88134df.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/5aff6ecadb439bb9c16a215188258f9474052fae.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/96642d7998cdd226b395153b7d8f3d36655c0176.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/743b4b30bf4fa11e3231dd0cc319016db1cb5b73.flv', 'VCDB_core_sample/saving_private_ryan_omaha_beach/283043f1c1433d9205416d2922764421d3d7e34e.flv']\n",
      "video VCDB_core_sample/david_beckham_lights_the_olympic_torch/09b682c899b0727e9990d8e347cdce3df7c5550e.flv's top 5 search result: ['VCDB_core_sample/david_beckham_lights_the_olympic_torch/09b682c899b0727e9990d8e347cdce3df7c5550e.flv', 'VCDB_core_sample/obama_kicks_door/df0c9e9664cfa6720c94e13eae35ddb7a9b5b927.flv', 'VCDB_core_sample/david_beckham_lights_the_olympic_torch/874a7d3cbfc17a46040b2d657667520bde309ca4.flv', 'VCDB_core_sample/run_forrest_fun/067fb2aa9623905a42a2b0b286de1386e45c5bf8.flv', 'VCDB_core_sample/david_beckham_lights_the_olympic_torch/5ac8651ce41781440c8595d174748f1a6cad1ff7.flv']\n",
      "video VCDB_core_sample/beautiful_mind_game_theory/46f2e964ae16f5c27fad70d6849c76616fad7502.flv's top 5 search result: ['VCDB_core_sample/beautiful_mind_game_theory/46f2e964ae16f5c27fad70d6849c76616fad7502.flv', 'VCDB_core_sample/beautiful_mind_game_theory/5f6527456bf2b4c460737a1c222dbd0e70009030.flv', 'VCDB_core_sample/beautiful_mind_game_theory/b41c61fb756014444b7974ef25f2c76d5c3b42e4.flv', 'VCDB_core_sample/beautiful_mind_game_theory/6171d3d87ae377e497199554033bca96a263277b.mp4', 'VCDB_core_sample/mr_and_mrs_smith_tango/9ac82ed37390f45bdc81b699c53085b29143ee70.flv']\n",
      "video VCDB_core_sample/osama_bin_laden_is_dead_obama_speech_at_white_house/53ff5686f46ac55ae5357e6e7856faf0f5f15173.flv's top 5 search result: ['VCDB_core_sample/osama_bin_laden_is_dead_obama_speech_at_white_house/53ff5686f46ac55ae5357e6e7856faf0f5f15173.flv', 'VCDB_core_sample/osama_bin_laden_is_dead_obama_speech_at_white_house/77cf367bdc30892faa364bab0006c316a4e86f8e.flv', 'VCDB_core_sample/mr_and_mrs_smith_tango/be51784c0b0676bf487376af362b088922efa954.flv', 'VCDB_core_sample/osama_bin_laden_is_dead_obama_speech_at_white_house/b65af0803a5dcaac7776b7c8648091ca3bec8471.mp4', 'VCDB_core_sample/obama_kicks_door/14c81d68b80d04743a107d4de859cb4724ccc2c1.flv']\n",
      "CPU times: user 37.7 s, sys: 8.33 s, total: 46 s\n",
      "Wall time: 40 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "collection.load()\n",
    "\n",
    "def average_precision(x, y):\n",
    "    sum = 0 \n",
    "    ret = []\n",
    "    for i, e in enumerate(y):\n",
    "        sum += (e==x)\n",
    "        ret.append(sum/(i+1))\n",
    "    return ret\n",
    "\n",
    "search_pipe = (\n",
    "    pipe.input('path', 'event', 'id')\n",
    "        .map('event', 'ground_truth_event', lambda x: x)\n",
    "        .flat_map('path', 'frames', ops.video_decode.ffmpeg(start_time=0.0, end_time=60.0, sample_type='time_step_sample', args={'time_step': 1}))\n",
    "        .window_all('frames', 'video', lambda x: x)\n",
    "        .map('video', 'emb', ops.video_copy_detection.distill_and_select(model_name='cg_student', device = device))\n",
    "\t\t.map('emb', 'topk_raw_res', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='video_deduplication', limit=min_sample_num, metric_type='IP'))\n",
    "        .map('topk_raw_res', 'topk_events', lambda res: [id2event[x[0]] for x in res])\n",
    "        .map('topk_raw_res', 'topk_path', lambda res: [id2path[x[0]] for x in res])    \n",
    "        .map(('ground_truth_event', 'topk_events'), 'AP', average_precision)    \n",
    "        .output('path', 'id', 'topk_events', 'topk_path', 'ground_truth_event', 'AP')\n",
    ")\n",
    "\n",
    "APs = []\n",
    "dc_list = []\n",
    "with open(query_df_csv_path, 'r') as f:\n",
    "    csv_reader = reader(f)\n",
    "    _ = next(csv_reader)\n",
    "    for n, row in enumerate(csv_reader):\n",
    "        res = search_pipe(*row[1:])\n",
    "        res_dict = res.get_dict()\n",
    "        dc_list.append(res_dict)\n",
    "        APs += res_dict['AP']\n",
    "        path = res_dict['path']\n",
    "        top_k_path = res_dict['topk_path']\n",
    "        print(f'video {path}\\'s top {min_sample_num} search result: {top_k_path}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a026d731",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Mean Average Precision at 5 is: 0.8873684210526316\n"
     ]
    }
   ],
   "source": [
    "print(f'The Mean Average Precision at {min_sample_num} is: {sum(APs) / len(APs)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2003b8d",
   "metadata": {},
   "source": [
    "We found that we achieved an excellent topk metric on this easy small dataset, which means that if we limit each event to have k duplicate videos, then they can all be almost recalled and they are almost true positive ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b44a40",
   "metadata": {},
   "source": [
    "## Show query results\n",
    "\n",
    "With all the milvus search result, we can take a look at the query and the results video for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6d461033",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query_video = VCDB_core_sample/obama_kicks_door/1f73466d86f0a92140bd2f89eb95e8c147b47935.flv\n",
      "predict_topk_video_list = ['VCDB_core_sample/david_beckham_lights_the_olympic_torch/09b682c899b0727e9990d8e347cdce3df7c5550e.flv', 'VCDB_core_sample/troy_achilles_and_hector/0b3f9e88e5ab73e19dc4d1a32115ea3457867128.flv', 'VCDB_core_sample/scent_of_woman_tango/d1a19730dcc2b5ea5bec61756c452772aae031dd.flv', 'VCDB_core_sample/troy_achilles_and_hector/ee417a6b882853ffcd3f78b380b0205a9411f4d6.flv']\n",
      "query_video = VCDB_core_sample/scent_of_woman_tango/ac1543adad8ecc32d98d9a67d75fb0a39f67b685.flv\n",
      "predict_topk_video_list = ['VCDB_core_sample/scent_of_woman_tango/f3c8c0c9b93e0a49d2508eee4aae618c1d69e082.flv', 'VCDB_core_sample/scent_of_woman_tango/c29e02d9e847bcde39ee667b454e4d67b22b105c.flv', 'VCDB_core_sample/scent_of_woman_tango/d1a19730dcc2b5ea5bec61756c452772aae031dd.flv', 'VCDB_core_sample/beautiful_mind_game_theory/6171d3d87ae377e497199554033bca96a263277b.mp4']\n",
      "query_video = VCDB_core_sample/beautiful_mind_game_theory/46f2e964ae16f5c27fad70d6849c76616fad7502.flv\n",
      "predict_topk_video_list = ['VCDB_core_sample/beautiful_mind_game_theory/5f6527456bf2b4c460737a1c222dbd0e70009030.flv', 'VCDB_core_sample/beautiful_mind_game_theory/b41c61fb756014444b7974ef25f2c76d5c3b42e4.flv', 'VCDB_core_sample/beautiful_mind_game_theory/6171d3d87ae377e497199554033bca96a263277b.mp4', 'VCDB_core_sample/mr_and_mrs_smith_tango/9ac82ed37390f45bdc81b699c53085b29143ee70.flv']\n"
     ]
    }
   ],
   "source": [
    "# dc_list = dc.to_list()\n",
    "sample_num = 3\n",
    "sample_idxs = random.sample(range(len(dc_list)), sample_num)\n",
    "def get_query_and_predict_videos(idx):\n",
    "    query_video = id2path[dc_list[idx]['id']]\n",
    "    print('query_video =', query_video)\n",
    "    predict_topk_video_list = dc_list[idx]['topk_path'][1:]\n",
    "    print('predict_topk_video_list =', predict_topk_video_list)\n",
    "    return query_video, predict_topk_video_list\n",
    "dsp_res_list = []\n",
    "for idx in sample_idxs:\n",
    "    query_video, predict_topk_video_list = get_query_and_predict_videos(idx)\n",
    "    show_video_list = [query_video] + predict_topk_video_list\n",
    "    caption_list = ['query video: ' + Path(query_video).parent.stem] + ['result{0} video'.format(i) for i in range(len(predict_topk_video_list))]\n",
    "    dsp_res_list.append(display_gifs_from_video(show_video_list, caption_list, tmpdirname=tmpdirname))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "486b42fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"tmp_gifs/1f73466d86f0a92140bd2f89eb95e8c147b47935.gif\"> query video: obama_kicks_door <br/><br/><img src=\"tmp_gifs/09b682c899b0727e9990d8e347cdce3df7c5550e.gif\"> result0 video <br/><br/><img src=\"tmp_gifs/0b3f9e88e5ab73e19dc4d1a32115ea3457867128.gif\"> result1 video <br/><br/><img src=\"tmp_gifs/d1a19730dcc2b5ea5bec61756c452772aae031dd.gif\"> result2 video <br/><br/><img src=\"tmp_gifs/ee417a6b882853ffcd3f78b380b0205a9411f4d6.gif\"> result3 video <br/><br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dsp_res_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b971fb56",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"tmp_gifs/ac1543adad8ecc32d98d9a67d75fb0a39f67b685.gif\"> query video: scent_of_woman_tango <br/><br/><img src=\"tmp_gifs/f3c8c0c9b93e0a49d2508eee4aae618c1d69e082.gif\"> result0 video <br/><br/><img src=\"tmp_gifs/c29e02d9e847bcde39ee667b454e4d67b22b105c.gif\"> result1 video <br/><br/><img src=\"tmp_gifs/d1a19730dcc2b5ea5bec61756c452772aae031dd.gif\"> result2 video <br/><br/><img src=\"tmp_gifs/6171d3d87ae377e497199554033bca96a263277b.gif\"> result3 video <br/><br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dsp_res_list[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "56e0526b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"tmp_gifs/46f2e964ae16f5c27fad70d6849c76616fad7502.gif\"> query video: beautiful_mind_game_theory <br/><br/><img src=\"tmp_gifs/5f6527456bf2b4c460737a1c222dbd0e70009030.gif\"> result0 video <br/><br/><img src=\"tmp_gifs/b41c61fb756014444b7974ef25f2c76d5c3b42e4.gif\"> result1 video <br/><br/><img src=\"tmp_gifs/6171d3d87ae377e497199554033bca96a263277b.gif\"> result2 video <br/><br/><img src=\"tmp_gifs/9ac82ed37390f45bdc81b699c53085b29143ee70.gif\"> result3 video <br/><br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dsp_res_list[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "549fb90f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import shutil\n",
    "# shutil.rmtree(tmpdirname)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "f7dd10cdbe9a9c71f7e71741efd428241b5f9fa0fecdd29ae07a5706cd5ff8a2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}